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Dear Sir: 

Prior to the examination of the above-identified patent application, please enter the 
following amendments and consider the following remarks. 

IN THE SPECIFICATION 

Replace the paragraph beginning at page 1, line 25, with: 

In an article [Stuart Geman and Donald Geman. Stochastic relaxation, Gibbs 
distributions, and the Bayesian restoration of images. IEEE Transactions on Pattern 
Analysis and Machine Intelligence, PAMI-6(6): 72 1-741, November 1984], a stereo 
matching method using Markov random fields and stochastic optimization methods, 
based on simulated annealing presented by S. Kirkpatrick et al., "Optimization by 
Simulated Annealing", Science, May 1983, pg. 671-680, is described. This has been 
further developed by others, for example, Geiger and Girosi using mean field theory. 
However, this class of methods is iterative in nature resulting in very long computational 
times that are not suitable for real time stereo matching. 
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Replace the paragraph beginning at page 2, line 3 , with: 

In an article [H. H. Baker and T. O. Binford. Depth from edge and intensity based 
stereo. In Proceedings of the International Joint Conference on Artificial Intelligence, 
page 631-636, Vancouver, Canada, 1981] and an article [Y. Ohta and T. Kanade. Stereo 
by intra- and inter-scan line search. IEEE Transactions on Pattern Analysis and Machine 
Intelligence, PAMI-7(2):139-154, March 1985], stereo matching methods based on 
dynamic programming (DP) and heuristic post-processing are described. In an article 
[Ingemar J. Cox, Sunita L. Hingorani, Satish B. Rao, and Bruce M. Maggs. A maximum 
likelihood stereo algorithm. Computer Vision and Image Understanding, 63(3):542-567, 
May 1996] and an article [Stan Birchfield and Carlo Tomasi. Depth discontinuities by 
pixel-to-pixel stereo. In Proceeding of the IEEE International Conference on Computer 
Vision, pages 1073-1080m, Bombay, India, 1998], single-level DP in discrete pixel 
oriented methods are described. In an article [Peter N. Belhumeur. A Bayesian approach 
to binocular stereopsis. International Journal of Computer Vision, 19(3):237-260, 1996], 
a more complex DP method with sub-pixel resolution is described. Though this class of 
methods is much faster than the Markov random field based ones, they do not scale well 
for parallel processing and are thus still unsuitable for real-time stereo matching. 

Replace the paragraph beginning at page 5, line 28 , with: 

An image input to the left and right cameras 10 and 1 1 is converted into digital 
signals in the form of pixels in the image processing unit 12 and one scan line of each 
image is provided to the SMC 13 in units of a pixel. After the scan line is fully provided 
to the SMC 13, disparity data is output in units of a pixel. The process in which a 
disparity is output is repeated for all scan lines of the pair of images in the same way. 
Therefore, only the process for processing a pair of scan lines will now be explained. 
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Replace the paragraph beginning at page 6, line 3 , with: 

As shown in FIG. 2, the SMC 13 contains a linear array of N identical processing 
elements 22 and two linear arrays, each of N/2 image registers 20 and 21. Here, N is a 
multiple of 2. 

Replace the paragraph beginning at page 6, line 25 , with: 

Therefore, half of the processing elements 22 and half of the image registers (20 or 
21) operate at each system clock cycle, beginning from the even-numbered processing 
elements 22 and right image registers 20. The processing step is controlled by read/write 
signal (F/B or R/W, hereinafter referred to as "R/W"). When an R/W signal line is in a 
high state, data is written and when the R/W signal line is in a low state, data is read. 

Replace the paragraph beginning at page 7, line 6 , with: 

In the last ClkO in the initializing process, after the first half of the data in the scan 
line of the right image is input to the processing elements 22, the first pixel in the scan 
line of the left image is input to the processing elements 22. At this time, registers inside 
each processing element 22 are set to an appropriate initial value. The initial value of the 
processing element 0 is '0' and the initial value of all the other processors is the maximum 
(or close to the maximum) possible value. Then, the processing process is continuously 
applied to all pixel data input at each system clock until data in the present scan line is all 
processed (ClkE is for the left image, and ClkO is for the right image). 
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IN THE CLAIMS 

Replace the indicated claims with: 

1. (Amended) A real-time stereo image matching system comprising: 

signal converting means for converting an image input from a first camera and a 
second camera into a digital signal; and 

image matching means for calculating a matching cost based on a pair of pixels in 
one scan line of the first and second digital image signals, tracing a decision value which 
determines a minimum matching cost, and outputting the decision value as an estimated 
disparity according to predetermined activation information. 

2. (Amended) The real-time stereo image matching system of claim 1, wherein 
the first camera and second camera have optical axes parallel to each other and have 
coplanar focal planes on. 

3. (Amended) The real-time stereo image matching system of claim 1, wherein, 
in the image matching means, the matching cost is calculated after occlusion information 
for pixels which do not match in the scan line is added to the pair of pixels. 

4. (Amended) The real-time stereo image matching system of claim 1, wherein 
the image matching means further comprises: 

first storage means for storing the digital image pixels from the first camera; 

second storage means for storing the digital image pixels from the second camera; 

processing means for outputting a estimated disparity using pixels input from the 
first and second storage means; and 

clock control means for providing a clock signal for controlling operations of the 
first and second storage means and the processing means. 



4 



In re Application of Jeong et al. 
Application No. Unassigned 

5. (Amended) The real-time stereo image matching system of claim 4, wherein 
the system includes N processing means, N/2 first storage means, and N/2 second storage 
means where N is an integer multiple of 2. 

7. (Amended) The real-time stereo image matching system of claim 5, wherein, 
among the N processing means, only a processing means that outputs the predetermined 
disparity is activated at one time and the remaining processing means are in high 
impedance states. 

10. (Amended) The real-time stereo image matching system of claim 4, wherein 
the clock control means outputs a first clock signal for even-numbered processors and the 
second storage means, and a second clock signal for odd-numbered processors and the 
first storage means. 

1 1 . (Amended) The real-time stereo image matching system of claim 4, wherein 
the processing means comprises: 

a forward processor for receiving a pixel of one scan line in the first storage means 
and the second storage means, and outputting a determined matching cost and a decision 
value; 

decision storage means for storing the decision value output from the forward 
processor; and 

a second processor for outputting a determined disparity, using the decision value 
output from the decision storage means, according to activation information. 

12. (Amended) The real-time stereo image matching system of claim 1 1 , 
wherein, when a write control signal is input from outside, the first processor operates, 
and when a read control signal is input from outside, the second processor operates. 

13. (Amended) The real-time stereo image matching system of claim 1 1 , wherein 
the decision storage means has a last-in first-out structure in which the decision value that 
is output last from the first processor is first input to the second processor. 
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14. (Amended) The real-time stereo image matching system of claim 11, wherein 
the first processor comprises: 

matching cost calculating means for calculating a matching cost, using a pixel of 
one line in the first storage means and the second storage means; 

first adding means for adding the calculated matching cost to the fed-back 
accumulated cost; 

comparing means for comparing the output of the first adding means with the 
costs of neighboring processing means, and then outputting the minimum matches of cost 
and decision value; 

storage means for storing the minimum cost that is a comparison produced by the 
comparison means, as the accumulated cost; and 

second adding means for adding the entire cost and occlusion cost to produce a 
sum, and then outputting the sum to neighboring processing means. 

1 5 . (Amended) The real-time stereo image matching system of claim 1 1 , wherein 
the second processor comprises: 

logical OR means for OR-ing activation information of the neighboring processing 
means and feed-back activation information route; 

a register for storing the last activation information produced by the logical OR 

means; 

demultiplexing means for demultiplexing the last activation information according 
to the decision value output from the decision storage means and outputting to the 
neighboring processing means and feeding back to the logical OR means; and 

a tri-state buffer for outputting the decision value output from the decision storage 
means, as a determined disparity, according to the activation information of the register. 
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IN THE ABSTRACT 

Replace the abstract with: 

Abstract 

A system for processing stereo matching of a video image sequence in a real-time 
mode. The system includes a signal converter for converting an image input from a first 
camera and a second camera into a digital signal; and an image matching chip for 
calculating a determined matching cost based on a pair of pixels in one scan line of the 
first and second digital image signals, tracing the decision value which determines the 
minimum matching cost, and outputting the decided value as an estimated disparity 
according to determined activation information; and a display for displaying the output 
from the image matching. According to the system, real-time stereo matching is enabled 
by parallel processing or video image sequences using an algorithm which is based on a 
new dynamic trellis based method and is optimal in the Bayesian sense. 
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REMARKS 



The foregoing amendments are made to correct minor translational errors and to 
meet United States requirements as to form. No new matter is added. 

Respectfully submitted, 




Suite 300 

700 Thirteenth Street, N. W. 
Washington, D. C. 20005 
Telephone: (202) 737-6770 
Facsimile: 1202) 737-6776 

JAW/cmcg 7 
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SPECIFICATION, CLAIMS AND 
ABSTRACT AS PRELIMINARILY AMENDED 

Amendments to the paragraph beginning at page 1, line 25: 

In an article [Stuart Geman and Donald Geman. Stochastic relaxation, Gibbs 
distributions, and the Bayesian restoration of images. IEEE Transactions on Pattern 
Analysis and Machine Intelligence, PAMI-6(6):721-741 ^feemfeeg November 1984], a 
stereo matching method using Markov random fields and stochastic optimization 
methods, based on simulated annealing presented by S. Kirkpatrick et al., "Optimization 
by Simulated Annealing", Science, May 1983, pg. 671-680, is described. This has been 
further developed by others, for example, Geiger and Girosi using mean field theory. 
However, this class of methods is iterative in nature resulting in very long computational 
times that are not suitable for real time stereo matching. 

Amendments to the paragraph beginning at page 2, line 3: 

In an article [H. H. Baker and T. O. Binford. Depth from edge and intensity based 
stereo. In Proceedings of the International Joint Conference o n Articifical Artificial 
Intelligence, page 631-636, Vancouver, Canada, 1981] and an article [Y. Ohta and T. 
Kanade. Stereo by intra- and inter-scan line search. IEEE Transactions on Pattern 
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Analysis and Machine Intelligence, PAMI-7(2): 139-154, March 1985], stereo matching 
methods based on dynamic programming (DP) and heuristic post-processing are 
described. In an article [Ingemar J. Cox, Sunita L. Hingorani, Satish B. Rao, and Bruce 
M. Maggs. A maximu m liklihoo d likelihood stereo algorithm. Computer Vision and 
Image Understanding, 63(3):542-567, May 1996] and an article [Stan Birchfield and 
Carlo Tomasi. Depth discontinuities by pixel-to-pixel stereo. In Proceeding of the IEEE 
International Conference on Computer Vision, pages 1073- 1080m, Bombay, India, 1998], 
single-level DP in discrete pixel oriented methods are described. In an article [Peter N. 
Belhumeur. A Bayesian approach to binocular stereopsis. International Journal of 
Computer Vision, 19(3):237-260, 1996], a more complex DP method with sub-pixel 
resolution is described. Though this class of methods is much faster than the Markov 
random field based ones, they do not scale well for parallel processing and are thus still 
unsuitable for real-time stereo matching. 

Amendments to the paragraph beginning at page 5, line 28: 

An image input to the left and right cameras 10 and 1 1 is converted into digital 
signals in the form of pixels in the image processing unit 12 and one scan line of each 
image is provided to the SMC 13 in units of a pixel. After the scan line is fully provided 
to the SMC 13, disparity data is output in units of a pixel. The process in which a 
disparity is output is repeated for all scan lines of the pair of images in the same way. 
Therefore, only the process for processing a pair ofa scan4» e lines will now be 
explained. 

Amendments to the paragraph beginning at page 6, line 3: 

As shown in FIG. 2, the SMC 13 contains a linear array of N identical processing 
elements 22 and two linear arrays, each of N/2 image registers 20 and 21 . Here, N is a 
multiple of 2. 
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Amendments to the paragraph beginning at page 6, line 25: 

Therefore, half of the processing elements 22 and half of the image registers (20 or 
21) operate a t ever y each system clock cycle, beginning from the even-numbered 
processing elements 22 and right image registers 20. The processing step is controlled by 
read/write signal (F/B or R/W, hereinafter referred to as "R/W"). When an RAV signal 
line is in a high state, data is written and when the R/W signal line is in a low state, data is 
read. 

Amendments to the paragraph beginning at page 7, line 6: 

In the last ClkO in the initializing process, after the first half of the data in the scan 
line of the right image is input to the processing elements 22, the first pixel in the scan 
line of the left image is input to the processing elements 22. At this time, registers inside 
each processing element 224s are set to an appropriate initial value. The initial value of 
the processing element 0 is '0' and the initial value of all the other processors is the 
maximum (or close to the maximum) possible value. Then, the processing process is 
continuously applied to all pixel data input at each system clock=as^jmtil data in the 
present scan line is all processed (ClkE is for the left image, and ClkO is for the right 
image). 

Amendments to the existing claims: 

1. (Amended) A real-time stereo image matching system comprising: 
^signal converting means for converting an image input from a first camera and a 
second camera into a digital signal; and 

& image matching means for calculating a determined matching cost based on a 
pair of pixels in one scan line of the first and second digital image signals, tracing=4he 
d#e#4@ d a decision value which determines4k#a minimum matching cost, and outputting 
the decided decision value as an estimated disparity according to predetermined 
activation information. 
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2. (Amended) The real-time stereo image matching system of claim 1, wherein 
the image input to the signal converting moans is obtained by the first camera and second 
camera=4hat have optical axes parallel to each other and have4fa e coplanar focal planes on 



3. (Amended) The real-time stereo image matching system of claim 1, wherein^ 
in the image matching means, the calculation of the matching cost is calculated after 
occlusion information's for pixels whic h pixels do not match in the scan line is added to 

4. (Amended) The real-time stereo image matching system of claim 1, wherein 
the image matching means further comprises: 

ft-first storage means for storing the digital image pixels from the first camera; 
^second storage means for storing the digital image pixels from the second 
camera; 

a=processing means for outputting a predetermined estimated disparity using pixels 
input from the firs t storage means and second storage means; and 

*=clock control means for providing a clock signal for controlling^ operations of 
the first and second storage means and the processing means. 

5. (Amended) The real-time stereo image matching system of claim 4, wherein 
the system in formed o f includes N processing means, N/2 first storage means, and N/2 
second storage means (IIero,j vhere N is= a an integer multiple of 2). 

7. (Amended) The real-time stereo image matching system of claim 5, wherein 
among the N processing means, only=feea processing means that outputs=athe 
predetermined disparity is activated at one time and the remaining processing means are 
in high impedance states. 
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10. (Amended) The real-time stereo image matching system of claim 4, wherein 
the clock control means outputs a first clock signal for=4he even-numbered processors and 
the second storage means, and a second clock signal fcn=fee odd-numbered processors and 
the first storage means. 

1 1 . (Amended) The real-time stereo image matching system of claim 4, wherein 
the processing means comprises: 

a forward processor for receiving a pixel of one scan line in the first storage means 
and the second storage means, and outputting a determined matching cost and a decision 
value; 

^decision storage means for storing the decision value output from the forward 
processor; and 

a second processor for outputting a determined disparity, using the decision value 
output from the decision storage means* according to the determined activation 
information. 

12. (Amended) The real-time stereo image matching system of claim 11, 
wherein,, when a write control signal is input from outside, the first processor operates, 
and when a read control signal is input from outside, the second processor operates. 

13. (Amended) The real-time stereo image matching system of claim 11, wherein 
the decision storage means has a last-in first-out structure in which the decided decision 
value that is output last from the first processor is first input to the second processor. 

14. (Amended) The real-time stereo image matching system of claim 1 1, wherein 
the first processor comprises: 

^matching cost calculating means for calculating a matching cost, using a pixel of 
one line in the first storage means and the second storage means; 

a=first adding means for adding the calculated matching cost to the fed-back 
accumulated cost; 
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^comparing means for comparing the output of the first adding means with the 
costs of neighboring processing means, and then outputting the minimum matches of cost 
and decision value; 

a=storage means for storing the minimum cost that is the rcoult o f a comparison 
produced by the comparison means , as the accumulated cost; and 

^second adding means for adding the entire cost and occlusion cost to produce a 
sum , and then outputting the result sum to neighboring processing means. 

15. (Amended) The real-time stereo image matching system of claim 1 1, wherein 
the second processor comprises: 

^logical OR means fo r performing OR-ing=fee activation information of the 
neighboring processing means and4fes feed-back activation information route; 

a register for storing the last activation informatio n and that io the rcoult of 
produced by the=QR-^ # logical OR means ; 

^demultiplexing means for demultiplexing the last activation information 
according to the decision value output from the decision storage means to output_ and 
outputting to the neighboring processing means and^ed feeding back to the logical OR 
means; and 

a tri-state buffer for outputting the decision value output from the decision storage 
means, as a determined disparity, according to the activation information of the register. 

Amendments to the abstract: 

Abstract 

A system for processing stereo matching of a video image sequence in a real-time 
mode is provided. The system includes a signal converting me ana converter for 
converting an image input from a first camera and a second camera into a digital signal; 
and an image matching=«esf^_chip for calculating a determined matching cost based on a 
pair of pixels in one scan line of the first and second digital image signals, tracing the 
decision value which determines the minimum matching cost, and outputting the decided 
value as an estimated disparity according to determined activation information; and a 
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displa y means for displaying the output from the image matching=3»esns. According to 
the system, real-time stereo matching is enabled? by parallel processing or video image 
sequences using an algorithm which is based on a new dynamic trellis based method and 
is optimal in the Bayesian sense. 
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1 . A real-time stereo image matching system comprising: 

signal converting means for converting an image input from a first camera and a 
second camera into a digital signal; and 

image matching means for calculating a matching cost based on a pair of pixels in 
one scan line of the first and second digital image signals, tracing a decision value which 
determines a minimum matching cost, and outputting the decision value as an estimated 
disparity according to predetermined activation information. 

2. The real-time stereo image matching system of claim 1 , wherein the first 
camera and second camera have optical axes parallel to each other and have coplanar 
focal planes on. 



3. The real-time stereo image matching system of claim 1, wherein, in the image 
matching means, the matching cost is calculated after occlusion information for pixels 
which do not match in the scan line is added to the pair of pixels. 
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4. The real-time stereo image matching system of claim 1, wherein the image 
matching means further comprises: 

first storage means for storing the digital image pixels from the first camera; 

second storage means for storing the digital image pixels from the second camera; 

processing means for outputting a estimated disparity using pixels input from the 
first and second storage means; and 

clock control means for providing a clock signal for controlling operations of the 
first and second storage means and the processing means. 

5. The real-time stereo image matching system of claim 4, wherein the system 
includes N processing means, N/2 first storage means, and N/2 second storage means 
where N is an integer multiple of 2. 

6. The real-time stereo image matching system of claim 5, wherein the processing 
means exchanges information with neighboring processing means. 

7. The real-time stereo image matching system of claim 5, wherein, among the N 
processing means, only a processing means that outputs the predetermined disparity is 
activated at one time and the remaining processing means are in high impedance states. 

8. The real-time stereo image matching system of claim 4, wherein 

the first storage means and the second storage means are initialized when the processing 
means completes processing of pixels in one scan line. 

9. The real-time stereo image matching system of claim 4, wherein a pixel stored 
in the first storage means is delayed by (N/2-1) clock cycles compared to a pixel stored in 
the second storage means. 
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10. The real-time stereo image matching system of claim 4, wherein the clock 
control means outputs a first clock signal for even-numbered processors and the second 
storage means, and a second clock signal for odd-numbered processors and the first 
storage means. 

1 1 . The real-time stereo image matching system of claim 4, wherein the 
processing means comprises: 

a forward processor for receiving a pixel of one scan line in the first storage means 
and the second storage means, and outputting a determined matching cost and a decision 
value; 

decision storage means for storing the decision value output from the forward 
processor; and 

a second processor for outputting a determined disparity, using the decision value 
output from the decision storage means, according to activation information. 

12. The real-time stereo image matching system of claim 11, wherein, when a 
write control signal is input from outside, the first processor operates, and when a read 
control signal is input from outside, the second processor operates. 

13. The real-time stereo image matching system of claim 11, wherein the decision 
storage means has a last-in first-out structure in which the decision value that is output 
last from the first processor is first input to the second processor. 

14. The real-time stereo image matching system of claim 1 1 , wherein the first 
processor comprises: 

matching cost calculating means for calculating a matching cost, using a pixel of 
one line in the first storage means and the second storage means; 

first adding means for adding the calculated matching cost to the fed-back 
accumulated cost; 
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comparing means for comparing the output of the first adding means with the 
costs of neighboring processing means, and then outputting the minimum matches of cost 
and decision value; 

storage means for storing the minimum cost that is a comparison produced by the 
comparison means, as the accumulated cost; and 

second adding means for adding the entire cost and occlusion cost to produce a 
sum, and then outputting the sum to neighboring processing means. 

15. The real-time stereo image matching system of claim 11, wherein the second 
processor comprises: 

logical OR means for OR-ing activation information of the neighboring processing 
means and feed-back activation information route; 

a register for storing the last activation information produced by the logical OR 

means; 

demultiplexing means for demultiplexing the last activation information according 
to the decision value output from the decision storage means and outputting to the 
neighboring processing means and feeding back to the logical OR means; and 

a tri-state buffer for outputting the decision value output from the decision storage 
means, as a determined disparity, according to the activation information of the register. 

16. The real-time stereo image matching system of claim 17, wherein the output 
from the decision storage means controls which direction the demultiplexing means 
passes the activation information. 
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